Current anti-virus/anti-malware technologies are used to monitor critical application and kernel code for introspection to track illegal usage by malware or virus. The technique for monitoring critical code (such as application programming interfaces (APIs)) involves hooking the critical code and enforcing a detour to the anti-virus agent for introspection before allowing the critical code execution to continue.
One method used for hooking/detour to the anti-virus agent is instruction patching the original code for detour to the anti-virus agent. However the disadvantage of this method that it is intrusive and non-transparent. As a result hooks can be easily detected by the virus. Also, patching the original code becomes complex due to instruction boundary issues. For example, it is not safe to patch when the size of the original instruction replaced by the patch is smaller than the size of the patch instruction. It is also not safe to patch instructions that have return instruction pointer (RIP)-relative addressing since the anti-malware agent code executes at a different virtual address in the monitored address space.
Another such method is by page permission revocation such as marking a page having critical code as non-executable and steering control to the anti-virus agent via exceptions. A disadvantage of this method is that it has high overhead. In addition, there is the performance impact of taking exceptions. Moreover there can be a high number of false positives due to coarse-grained (e.g., 4 kilobyte (kB) page-level) hooking, which causes unnecessary context switches to the anti-virus agent for analysis.